Automatic face and skin beautification using face detection

ABSTRACT

Sub-regions within a face image are identified to be enhanced by applying a localized smoothing kernel to luminance data corresponding to the sub-regions of the face image. An enhanced face image is generated including an enhanced version of the face that includes certain original pixels in combination with pixels corresponding to the one or more enhanced sub-regions of the face.

PRIORITY AND RELATED APPLICATIONS

This application is a continuation application that claims the benefitof priority to U.S. patent application Ser. No. 12/512,796, filed Jul.30, 2009, now U.S. Pat. No. 8,345,114; which claims priority to U.S.provisional patent application No. 60/084,942, filed Jul. 30, 2008,which are incorporated by reference. The '796 application is one ofthree applications filed contemporaneously by these same inventors.

BACKGROUND

1. Field of Invention

The invention relates to image processing, particularly of detectedsub-regions within face images.

2. Description of the Related Art

Proctor and Gamble's U.S. Pat. No. 6,571,003 mentions finding and fixingfacial defects such as spots, wrinkles, pores, and texture insub-regions of faces, e.g, cheeks or in areas defined by landmark pointssuch as corner or nose, eye, or mouth. The technique involves replacingthe defined region with a mask. The P&G patent discloses toelectronically alter the color.

The P&G patent also mentions detecting and correcting lighting gradientsand lighting variances. These lighting gradients, or variances, appearto involve instances where there is directional lighting which may causea sheen or brighter region on the facial skin. U.S. patent applicationsSer. Nos. 12/038,147, 61/106,910 and 61/221,425, which are assigned tothe same assignee as the present application and are hereby incorporatedby reference, describe techniques which use Viola-Jones type classifiercascades to detect directional lighting. However, determining andcorrecting a lighting gradient would typically involve global analysis,exceptions being possible in combination with face-tracking techniquessuch as those described at U.S. patents Ser. Nos. 7,403,643 and7,315,631 and U.S. applications Ser. Nos. 11/766,674, published as2008/0037840, and U.S. Ser. Nos. 12/063,089, 61/120,289, and 12/479.593,which are all assigned to the same assignee as the present applicationand are hereby incorporated by reference. It is desired to have atechnique that uses a local blurring kernel rather than such techniquesinvolving less efficient global analysis for certain applications and/orunder certain conditions, environments or constraints.

Kodak's U.S. Pat. No. 7,212,657 illustrates at FIGS. 13-14 to generate ashadow/peak image (based on generating a luminance image and an averageluminance image), a blur image, and blended images. The Kodak patentstates that a shadow/highlight strength image is generated bysubtracting an average luminance image from a luminance image. Also, atFIG. 16, the Kodak patent shows element 1530 is labeled as “generateluminance and chrominance scaling factors using peak/valley map andcolor info”, and element 1540 is labeled as “modify luminance andchrominance of pixels within mask regions”. Face detection is describedin the Kodak patent, but not face tracking.

The Kodak technique, like the P&G technique, involves global imagemanipulations, i.e., the “luminance image” is not indicated as includinganything less than the entire image, the “blur image” involves theapplication of a kernel to the entire image, and the “blended image”involves three copies of the global image. The “blur image” involveschrominance and luminance data meaning that a lot of memory is used formanipulating the image, particularly if the application involves aresource constrained embedded system. Regarding luminance andchrominance scaling factors, even if they involve localized scalingfactors, they are not described in the Kodak patent as being generatedfor application to anything less than the entire image.

U.S. patent application Ser. Nos. 11/856,721 and 12/330,719, which areassigned to the same assignee as the present application and are herebyincorporated by reference, describes a technique that can be applied asa single, raster-like, scan across relevant regions of an image withoutinvolving global analysis or a determination of global properties suchas the average luminance image, or a shadow or blur image. Suchsingle-pass scan through predetermined regions provides a far moreefficient and suitable technique for embedded systems such as digitalcameras than either of the P&G or Kodak patents.

The Hewlett Packard (HP) published patent application 2002/0081003mentions air-brushing which typically involves applying color over aswath of an image, e.g., such as may include a blemish or wrinkle. TheHP publication also mentions blurring over a wrinkle on an image of aperson's face, and again specifically describes blurring or blendingcolor values defining the wrinkles and surrounding skin. The HPapplication mentions changing brightness to brighten or darken a facialfeature, such as to shade a facial feature, and goes on to describechanging color values of skin associated with the feature to shade thefeature. The HP patent further discloses to sharpen a hair line and/orblur a forehead and/or cheeks, by blurring color values. Face detectionand face tracking over multiple images, full resolution or lowresolution and/or subsample reference images such as previews, postviewsand/or reference images captured with a separate imaging system before,during or after capturing of a main full-resolution image are notdescribed in the HP patent, nor is there any suggestions to smooth orblur luminance data of a digital face image.

Portrait is one of the most popular scenes in digital photography. Imageretouching on portrait images is a desirable component of an imageprocessing system. Users can spend a lot of time with conventionalsoftware trying to make a portrait nicer by hiding wrinkles andblemishes. It is desired to provide an innovative automatic portraitscene enhancer, which is suitable for an embedded device, such as adigital still camera, camera-phone, or other handheld or otherwiseportable consumer appliance having image acquisition components (e.g.,lens, image sensor) and a processor.

SUMMARY OF THE INVENTION

A method is provided for enhancing an appearance of a face within adigital image using a processor. An image is acquired of a sceneincluding a face. The face is identified within the digital image. Oneor more sub-regions to be enhanced with localized luminance smoothingare identified within the face. One or more localized luminancesmoothing kernels are applied each to one of the one or more sub-regionsidentified within the face to produce one or more enhanced sub-regionsof the face. The one or more localized smoothing kernels are applied toluminance data of the one or more sub-regions identified within theface. An enhanced image is generated including an enhanced version ofthe face including certain original pixels in combination with pixelscorresponding to the one or more enhanced sub-regions of the face. Theenhanced image and/or a further processed version is displayed,transmitted, communicated and/or digitally stored and/or otherwiseoutput.

The localized luminance smoothing may include blurring or averagingluminance data, or a combination thereof.

One or more localized color smoothing kernels may be applied to the oneor more sub-regions. The one or more enhanced sub-regions of thecorrected image may also include pixels modified from original orotherwise processed pixels of the face at least by localized colorsmoothing.

Noise reduction and/or enhancement may be applied to the one or moresub-regions. The one or more enhanced sub-regions of the corrected imagemay also include pixels modified from original or otherwise processedpixels of the face at least by localized noise reduction and/orenhancement.

Certain non-skin tone pixels within the one or more sub-regions of theface may be determined not to have a threshold skin tone. These non-skintone pixels may be removed, replaced, reduced in intensity, and/ormodified in color.

Enhanced pixels of the one or more enhanced sub-regions may includeenhanced intensities which comprise one or more functions of arelationship between original pixel intensities and local averageintensities within the one or more original and/or enhanced sub-regions.

One or more mouth and/or eye regions may be detected within the face. Anatural color of one or more sub-regions within the one or more mouthand/or eye regions may be identified and enhanced. These sub-regions mayinclude one or more teeth, lips, tongues, eye whites, eye brows, iris's,eye lashes, and/or pupils.

The face may be classified according to its age based on comparing oneor more default image attribute values with one or more determinedvalues. One or more camera acquisition and/or post-processing parametersmay be adjusted based on the classifying of the face according to itsage.

A digital image acquisition device is also provided, including a lens,an image sensor and a processor, and a processor-readable memory havingembodied therein processor-readable code for programming the processorto perform any of the methods described herein, particularly forenhancing an appearance of a face or other feature within a digitalimage.

One or more processor-readable media are also provided that haveembodied therein code for programming one or more processors to performany of the methods described herein.

In certain embodiments, face tracking using previews, postviews or otherreference images, taken with a same or separate imaging system as a mainfull resolution image is combined with face beautification. Thisinvolves smoothing and/or blurring of face features or face regions,wrinkle/blemish removal, or other digital cosmetic adjustments. Incertain embodiments, a luminance channel is used for smoothing anunsightly feature, while in a narrow subset of these, only the luminancechannel is used for smoothing without using any color channel. Otherembodiments used one or more color channels in addition to the luminancechannel, and these may or may not also use face tracking.

In certain embodiments, localized modification of a region of a face isperformed based on an average of the pixel values surrounding aparticular pixel. This localized averaging/blurring kernel may beapplied solely on the luminance channel, thereby reducing computation inan embedded system such as a portable digital camera, camera-phone,camera-equipped handheld computing device, etc.

A single-pass filtering kernel may be configured to act only on localluminance values within pre-determined regions of the image, and may becombined with a binary skin map. This is far more efficient, using lessmemory and executing more quickly, within an embedded imaging systemsuch as a digital camera.

Blurring or shading may be achieved by changing selected luminancevalues of one or more sub-regions of a face. An embodiment involvesapplying or subtracting luminance over a swath of an image, e.g., suchas may include a blemish or wrinkle. Blurring may also be applied to afacial feature region that includes a wrinkle on an image of a person'sface. Blurring and/or blending luminance values of a face featureregion, e.g., temple region, side of nose, forehead, chin, cheek region)defining the wrinkles and surrounding skin. Brightness may be changed tobrighten or darken a facial feature, such as to shade a facial feature,and this may be achieved by changing luminance values of skin associatedwith the feature to shade or brighten the feature.

In certain embodiment, a technique is provided including in-cameraprocessing of a still image including one or more faces as part of anacquisition process. The technique includes identifying a group ofpixels including a face within a digitally-acquired still image on aportable camera. One or more first processing portions of the image isdetermined including the group of pixels (the first portion may becharacterized as foreground). One or more second processing portions ofthe image other than the group of pixels is then determined (and may becharacterized as background). The technique may include automaticallyin-camera processing the first processing portion with a determinedlevel of smoothing, blurring, noise reduction or enhancement, or otherskin enhancement technique involving one or more luminance components ofthe pixels, while applying substantially less or no smoothing, blurring,noise reduction or enhancement or otherwise to the second processingportion to generate a processed image including the face. The processedimage or a further processed version including the face is then stored,displayed, transmitted, communicated, projected or otherwise controlledor output such as to a printer, display other computing device, or otherdigital rendering device for viewing the in-camera processed image. Themethod may include generating in-camera, capturing or otherwiseobtaining in-camera a collection of low resolution images including theface, and determining the first processing portion including analyzingthe collection of low resolution images. The analyzing may includetracking the face within the collection of low resolution images.

A further method is provided for enhancing an appearance of a facewithin a digital image. A digital image of a scene including a face isacquired using a processor. The image is captured using a lens and animage sensor, and/or the image is received following capture by a devicethat includes a lens and an image sensor. The face is identified withinthe digital image. Skin tone portions of the face are segmented fromface features including one or two eyes or a mouth or combinationsthereof. Within the skin tone portions of the face, one or more blemishregions that vary in luminance at least a threshold amount fromnon-blemished skin tone portions are identified. Luminance data of theone or more blemish regions is smoothed to generate smoothed luminancedata. An enhanced image is generated including an enhanced version ofthe face that has original luminance data of the one or more blemishregions replaced with the smoothed luminance data and combined withoriginal non-blemished skin tone portions. The enhanced image and/or afurther processed version is/are displayed, transmitted, communicated,digitally stored and/or otherwise output.

The localized luminance smoothing may include blurring and/or averagingluminance data.

The method may include applying one or more localized color smoothingkernels to the one or more sub-regions. The one or more enhancedsub-regions of the corrected image further may include pixels modifiedfrom original pixels of the face at least by localized color smoothing.

The method may include applying noise reduction or enhancement, or both,to the one or more sub-regions. The one or more enhanced sub-regions ofthe corrected image may include pixels modified from original pixels ofthe face at least by localized noise reduction and/or enhancement.

The method may include determining certain non-skin tone pixels withinthe one or more sub-regions that do not comprise a threshold skin tone,and removing, replacing, reducing an intensity of, or modifying a colorof said certain non-skin tone pixels, or combinations thereof.

Enhanced pixels of the one or more enhanced sub-regions may includeenhanced intensities which comprise one or more functions of arelationship between original pixel intensities and local averageintensities within the one or more original and/or enhanced sub-regions.

The method may include detecting one or more mouth and/or eye regionswithin the face, and identifying and enhancing a natural color of one ormore sub-regions within the one or more mouth or eye regions, includingone or more teeth, lips, tongues, eye whites, eye brows, iris's, eyelashes, or pupils, or combinations thereof.

A further method is provided for enhancing an appearance of a facewithin a digital image. A processor is used to generate in-camera,capture or otherwise obtain in-camera a collection of one or morerelatively low resolution images including a face. The face isidentified within the one or more relatively low resolution images. Skintone portions of the face are segmented from face features including oneor two eyes or a mouth or combinations thereof. Within the skin toneportions of the face, one or more blemish regions are identified thatvary in luminance at least a threshold amount from the skin toneportions. A main image is acquired that has a higher resolution than theone or more relatively low resolution images. The main image is capturedusing a lens and an image sensor, or received following capture by adevice that includes a lens and an image sensor, or a combinationthereof. The method further includes smoothing certain original data ofone or more regions of the main image that correspond to the same one ormore blemish regions identified in the relatively low resolution imagesto generate smoothed data for those one or more regions of the mainimage. An enhanced version of the main image includes an enhancedversion of the face and has the certain original data of the one or moreregions corresponding to one or more blemish regions replaced with thesmoothed data. The enhanced image and/or a further processed versionis/are displayed, transmitted, communicated and/or digitally stored orotherwise output.

The method may include tracking the face within a collection ofrelatively low resolution images.

The smoothing may include applying one or more localized luminancesmoothing kernels each to one of the one or more sub-regions identifiedwithin the face to produce one or more enhanced sub-regions of the face.The one or more localized luminance smoothing kernels may be applied toluminance data of the one or more sub-regions identified within saidface. The localized luminance smoothing may include blurring and/oraveraging luminance data. The method may also include applying one ormore localized color smoothing kernels to the one or more sub-regions.The one or more enhanced sub-regions of the corrected image may includepixels modified from original pixels of the face at least by localizedcolor smoothing.

The method may also include applying noise reduction and/or enhancementto the one or more sub-regions. The one or more enhanced sub-regions ofthe corrected image may include pixels modified from original pixels ofthe face at least by localized noise reduction and/or enhancement.

Certain non-skin tone pixels may be determined within the one or moresub-regions that do not comprise a threshold skin tone. The method mayinclude removing, replacing, reducing an intensity of, and/or modifyinga color of such non-skin tone pixels.

Enhanced pixels of the one or more enhanced sub-regions may includeenhanced intensities which have one or more functions of a relationshipbetween original pixel intensities and local average intensities withinthe one or more original and/or enhanced sub-regions.

One or more mouth and/or eye regions may be detected within the face. Anatural color may be identified and enhanced for one or more sub-regionswithin the one or more mouth and/or eye regions, including one or moreteeth, lips, tongues, eye whites, eye brows, iris's, eye lashes, and/orpupils.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A-1B illustrate unprocessed and processed images of a face, wherethe processing involves applying selective smoothing or blur to regionsof the face.

FIGS. 2A-2C illustrate identification of regions of a face, andprocessed and unprocessed version of a face image, wherein theprocessing involves application of selective smoothing or blurring ofcertain regions of the face.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Using at least one reference image, and in certain embodiments more thanone reference image, including a face region, the face region isdetected. In those embodiments wherein multiple reference images areused, a face region is preferably tracked. Face detection and trackingare performed preferably in accordance with one or more techniquesdescribed in the US patents and US patent applications listed above andbelow and which are incorporated by reference here.

Given an input image and one or more, preferably two or more, smaller,subsampled, and/or reduced resolution versions of the input image (e.g.,one QVGA and one XGA), the position of a face and of the eyes of theface within the input image are determined using face detection andpreferably face tracking. FIG. 1A shows an example of an unprocessedimage including a face, or at least an image of a face with originalimage data or image data that has been otherwise processed than byselective localized smoothing or blurring such as is described withreference to embodiments herein. The face beautification method appliesa selective blur, and/or other smoothing such as localized averaging oraccording to one or more of the methods specifically described below,which enhances the skin, e.g., softening and/or reducing wrinkles and/orspots. FIG. 1A illustrates an unprocessed image of a face beforeapplying selective smoothing. FIG. 1B illustrates a processed version ofthe image of the face of FIG. 1A, i.e., after applying selectivesmoothing to certain sub-regions of the face.

In an exemplary embodiment, the method may be performed as follows.Certain sub-regions of the face are identified, e.g., rectangularsub-regions or other polygonal or curved or partially-curved sub-regionswith or without one or more cusps or otherwise abrupt segmentalintersections or discontinuities. These sub-regions may be places whereit will be desired to apply selective smoothing, or these sub-regionsmay be those places outside of which it is desired to apply theselective smoothing, or a combination of these. For example, threesub-regions such as two eyes and a mouth may be identified for notapplying selective smoothing, and/or four sub-regions such as aforehead, two cheeks and a chin may be specifically selected forapplying localized luminance smoothing.

Now, in the embodiment where the two eye and mouth are identified, theskin around these facial sub-regions/rectangles is detected. This caninclude in certain embodiments creating a binary skin image, includingsegmenting the QVGA version of the image. In one embodiment, thisinvolves thresholding done in YCbCr.

A larger rectangle or other shape may be defined around the face as awhole. That is, outside of this larger facial shape, it may be desiredin most embodiments herein not to apply the selective smoothing(although there may be other reasons to smooth or blur a background orother region around a detected face in a digital image, such as to blura background region in order to highlight a face in the foreground; see,e.g., U.S. Pat. Nos. 7,469,071 and 12/253,839, which are assigned to thesame assignee and are hereby incorporated by reference). A skin map maybe filtered by morphological operations. The largest regions inside theface may be selected to be kept, and regions may be selected based onother criteria such as overall luminance, a certain threshold luminancecontrast such as may be indicative of wrinkled skin, a colorqualification such as a certain amount of red, a spotty texture, oranother unsatisfactory characteristic of a region or sub-region of aface. Lip detection may be performed based on color information (Crcomponent) and/or on the position of the eyes, nose and/or ears or otherface feature such as chin, cheeks, nose, facial hair, hair on top ofhead, or neck, and/or on a shape detector designed for specificallydetecting lips.

The skin inside of one or more face regions, not including the eye andmouth regions, is corrected. In certain embodiments this involves skinpixels from inside a face region having their luminance componentreplaced with different luminance values, such as an average value ofits neighbors, e.g., substantially all or a fair sampling of surroundingskin pixels, or all of a majority of pixels from one direction as if thepixels were being replaced by blurred pixels caused by relativecamera-object movement in a certain direction. Smoothing can include anaveraging process of skin pixels from other regions of the face, and/orcan be a calculation other than averaging such as to prioritize certainpixels over others. The prioritized pixels may be closest to the pixelbeing replaced or may have a color and/or luminance with greatercorrelation to a preferred skin tone.

Certain criteria may be applied as requirement(s) for correcting aregion within an image. For example, it may be set as requisite that theregion be inside a face, although alternatively the skin of a person'sneck, leg, arm, chest or other region may be corrected. It may be set asrequisite that the luminance component be within a certain range. Thatrange may depend on an average luminance of the skin within the certainface or a preferred luminance or a selected luminance. The certain pixelmay be selected or not selected depending on its relation with otherdetails within the face (e.g., eyes, nose, lips, ears, hair, etc.). Thenumber of neighbors used when modifying the current pixel (i.e., thekernel size) may be varied depending on the size of the face versus thesize of the image, or on a standard deviation of luminance values,and/or other factors may be taken into account such as the resolution ora determination as to how much fixing the particular face region orsub-region ought to receive. If the face is too small compared to theimage (e.g., the face uses below a threshold percentage of the availablepixel area, then the system can be set to apply no correction ofwrinkles, spots, etc., because such undesired features may not bevisible anyway. The averaging or other smoothing or blurring may be doneon a XGA image in order to improve speed.

Localized Blurring/Smoothing Kernel(s)

The blurring kernel or smoothing kernel in certain embodiments may bechanged, adjusted, selected, and/or configured based on one or morefactors specific to the image and/or group of images based upon which acorrected image is to be generated. A factor may be relative size of thefacial image to that of the main picture. Other factors may includeresolution of the face region and/or the entire image, processingcapacity and/or RAM or ROM capacity, and/or display, projection ortransmission capacity of an embedded device or processing or renderingenvironment with which the image is acquired, processed and/or output.

The blurring kernel may include a table, formula, calculation and/orplot of face sizes (e.g., 5% of image, 10% of image, 20% of image, etc)versus kernel sizes (e.g., 3×3, 4×4, 5×5, etc.) The kernel may also beadjusted based the relative location of the sub-region within a face.The kernel applied to the cheeks may be configured to blur cheekseffectively, while a different kernel to apply to the skin around theeyes may be configured to blur/smooth that skin most effectively, samefor the skin in the forehead, the skin around the mouth/chin, etc. Adifferent kernel can be applied to a bearded region or other hair regionor no smoothing may be applied to such regions. In a specific, simpleexample embodiment, the blurring/smoothing kernel is smaller when facesare smaller (two or more levels or one or more thresholds may be used).The blurring kernel may decrease working around eyes or lips or nose orbearded regions or low luminance regions or dark colored regions. Theblurring kernel may depend on average luminance around the point ofinterest.

The method in certain embodiments may include the application ofselective skin enhancement and/or noise removal. This provides analternative approach to determining the facial regions when abeautification filter or blurring/smoothing kernel might not be applied.

An Alternative Implementation: Lee-Based Filtering

A face beautifier may use certain relevant data gathered in a facetracking technique as described in reference cited herein andincorporated by reference (see below). That information may include aposition of the face and/or a feature within the face such as one orboth eyes, mouth or nose, information relating to where skin is detectedand its tone, luminance, shaded areas, direction relative to incominglight, etc. That data can also include the Cb,Cr,Y range within the facearea, and/or backlighting image information.

Application to Luminance Channel

The technique according to certain embodiments may employ modificationsof the luminance channel to achieve the filtering of the skin. Datarelating to variance within the luminance channel may also be used, andtexture information of the skin of the face region or sub-region may beused. Such texture information may include certain chrominance data, butmay also include only luminance data which defines such texture withinthe image. The variance on luminance may be utilized when selectingand/or performing blurring/smoothing, and may be applied specifically toseparating wrinkles (which are typically rather isolated) from thetexture of the face of a shaved man or even an unshaved man (wherevariance is high). The texture information may involve a measure of towhat degree areas or sub-regions are uniform or not. The textureinformation may include a recognized or learned or newly-analyzedpattern, which can be analyzed either on the luminance channel onlyand/or also on one or more color channels.

In certain embodiments, only face and eyes may be mandatory, while inothers certain other features may be required. Face tracking may be usedbut is not required for the technique to provide tremendous advantage inbeautifying a face. The location of a face within an image may begathered using face detection only or using face tracking. A dynamicskin-map and/or contrast info may be gathered using face tracking.

Within a digital camera or real-time imaging appliance, a real-time facetracking subsystem (operable on a sequence of preview, postview or otherreference images independent of the main image) may be operated, and onacquisition of a main image, facial enhancements may be performed basedon (i) an analysis of the facial region in the main acquired image and(ii) an analysis of face region metadata determined from the real-timeface tracking subsystem.

Facial Image Enhancement

Apart from the image to be enhanced, the algorithm may use (ifavailable) extra information, including the position of the face(s) andeyes in the given image which will help limiting the area of search, andtwo resized copies of the initial image (e.g.: one QVGA and one XGA).These two images may be used for faster processing power where accuracyis less critical.

An example algorithm according to certain embodiments may be describedas follows:

Enhancement Map Detection

Based on face information, skin tones similar to those inside a facerectangle are sought in the entire image. In detail, for each facepassed, the steps may be as follows in one example embodiment (notnecessarily in the order discussed below):

Compute the average saturation for the region of interest (entire facerectangle or other shape in this case). To avoid problems in cases suchas side illumination, the average saturation for the entire image mayalso computed and the minimum between the two may be used.

The relevant skin tone information (from the face rectangle) isextracted. This is done by geometrical considerations (and additionallyby color filtering). In one implementation this means: top, left andright of the rectangle are changed in such a way that ⅕ of each side isnot taken into account. Bottom (based on image orientation) stays thesame or not depending on whether it is deemed important to have the neckincluded. One implementation of color filtering may be the eliminationor reduction of luminance or change of color of pixels which aredetermined to have non-skin tones (e.g. blue pixels).

PCA (Principal Component Analysis) procedure may be applied on remainingpixels. A pixel may be given by a triplet. The covariance matrix of thegiven pixels is computed. The eigenvectors and eigenvalues of thecovariance matrix are then found. The three resulting eigenvectorsrepresent the axes of a new 3D coordinate system. The two leastimportant axes (corresponding to the two smallest eigenvalues) arefurther considered.

The coordinates of all inspected pixels on the two abovementioned axesare computed. The two histograms of the absolute value of thecoordinates are then computed: one histogram for each axis. For each ofthe two histograms, an acceptance threshold may be determined, forexample, using the following procedure. The corresponding cumulativehistogram H is computed. The threshold is taken such as to delimit agiven percentage of the total number of pixels (i.e., threshold Th istaken such as H(Th)˜=p %, with p being a predefined value). By choosingdifferent values for p one can vary the strength of the skin filtering.For example values taken for p may vary from 90.0% (for strongfiltering) up to 97.5% (for permissive filtering).

Compute the coordinates of each image pixel on the two axes resultingafter the PCA step and check if the absolute values are smaller than thethresholds obtained in the previous step.

For a pixel to be considered skin type further verification may be done.An example is to check that saturation is large enough in the YUV colorspace. Based on the average saturation computed in the first stage, eachpixel may be verified to have at least one of the U and V values largeenough. Also the luminance level of the pixel is checked to be in apredefined gamut. This is because we do not want to beautify dark hairor too bright areas where color information is not reliable.

In the same time a generic skin detection algorithm (e.g. simplethresholding on the YUV space) may be applied on the entire image toobtain a less reliable but more inclusive skin map. The role of thegeneric skin map may be manifold, as it may replace the PCA skin map incases where face information is not present. The skin map may also usedto improve the PCA skin map by helping in deciding if holes in the mapare going to be filled. The skin map may add up to the PCA skin map“uncertain skin pixels”, or pixels with a lower confidence which are tobe treated separately by the correction block.

The skin map may now be cleaned up by applying spatial filtering such asmorphological operations. At this point the skin map may have two levelsof confidence: PCA skin (high confidence) and uncertain skin (lowconfidence). The number of levels of confidence may be further increasedby taking into consideration the spatial positioning of a skin pixelinside the skin area. In one implementation, the closer one pixel is tothe interior of the map, the higher its confidence is set. In anotherimplementation, the number of skin confidence levels could be increasedfrom the PCA thresholding stage by using multiple thresholding of pixelcoefficients on the PCA axes.

Enhancement Map Correction

The skin pixels from inside the faces (or the ones from regions thatpassed skin filtering when no face is present) are corrected. An exampleprocess for performing this correction is described below.

A weight α ε [0,1]_(α) may be computed for each pixel describing howmuch correction it will receive. The higher the value of α, the morecorrection will be applied to that pixel. The weight may be based on thelocal standard-deviation computed on the XGA intensity image over asquared neighborhood (e.g. 16×16 for large-size skin areas, or 8×8 formedium-sized skin areas), but may also take into account other factors(e.g., the skin level of confidence, the proximity of the pixel to facefeatures, such as eyes and mouth etc.)

Initially, α is computed as:

${\alpha = \frac{\sigma_{skin}}{\sigma_{local}}},$where σ_(skin) is the standard deviation computed over the whole skinarea, while σ_(local) is the local tan deviation. Then α is limited to1.

α may be increased by a predefined factor (e.g., 1.1-1.25) for pixelshaving higher confidence of skin.

α may be decreased by a predefined factor for pixels located in thevicinity of face features, such as eyes and mouth (see FIG. 1). (For eyeand mouth detection, see chapter on eye and mouth beautification).

Special attention may be given to pixels located near the skin border.In this example, for those pixels, σ_(local) is higher owing to the factthat there is a strong edge in the computing neighborhood. In thesecases, the direction of the edge is sought (only the four maindirections are considered) and, based on it, the most uniform sub-windowof the current window is used for recomputing α and the local average.

α may also modified based on the relationship between the intensity ofthe current pixel and the local average (computed over the sameneighborhood as σ_(local)). This is because face artifacts that areattempted to be eliminated by face beautification (e.g, freckles,pimples, wrinkles) may be typically darker than skin, but not very dark.

In one embodiment, the following modification may be performed: if thecurrent intensity is greater than the local average, decrease α (highintensity, therefore, strongly reduce correction). If the currentintensity is much lower than the local average, decrease α (too dark tobe a face artifact, strongly reduce correction). If the currentintensity is lower than the local average, but the difference betweenthe two is small, increase α (very likely face artifact, thereforeincrease correction). If the current intensity is lower than the localaverage, and the difference between them is important, slightly decreaseα (less likely to be a face artifact, therefore slightly reducecorrection).

Apply correction on the intensity value, based on the relation:NewIntensity=α·LocalAverage+(1−α)·OldIntensity

The averaging may be computed on the same intensity image used for theweighting map (XGA image). This improves speed without affectingquality.

FIGS. 2A-2C illustrates an example of working with detected features. InFIG. 2A, input and predetermined data are illustrated with colorsincluding cyan (blue-ish hue) for the face rectangle, green for facefeatures such as eye and mouth or lips, and red for skin inside the facearea.

FIG. 2B illustrates an initial image, and FIG. 2C illustrates an outputresult using auto face beautification.

Enhancement of Facial Features (Eyes and Mouth)

Besides removing skin artifacts (wrinkles, pimples etc.), eyes and mouthbeautification may be applied as well towards an overall better visualaspect of the face. The following actions may be taken for eye and mouthbeautification.

Initial locations of eyes and mouth may be (coarsely) determined as thelargest holes in the PCA skin map located in the upper left, upper rightand lower half parts of the face rectangle or other shape.

More precise eye and mouth localization may be performed at a higherresolution (XGA at least) in a small neighborhood surrounding theinitial areas described above, as follows:

A mouth area may be detected based on color information. When using YUVcolor space, it may be defined as the area which has the V componenthigher than a threshold (computed based on the local V histogram).

The presence of teeth may be checked by inspecting the histogram ofsaturation S inside the smallest rectangle surrounding the mouth area.If working in YUV color space, saturation may be computed asS=abs(U)+abs(V). If the histogram of saturation is unimodal, then teethmight not be visible. If the histogram of saturations is bimodal, thenthe area corresponding to the inferior mode of the histogram may beinspected. If this area is found to be located inside the mouth area(more precisely, if a sandwich mouth-teeth-mouth is present), then itmay be decided that teeth are visible.

One or both eye areas may be detected each as a connected area that hasthe normalized Y·S component lower than a threshold (computed based onthe local Y·S histogram). In the above expression, Y is the normalizedintensity component from the YUV color space, whereas S is thenormalized saturation, computed as above. Normalization of both Y and Smay be done with respect to the local maximum values.

The iris may be detected as the central, darker part of the eye, whereassclera (eye white) may be detected as the remaining part of the eye.

Mouth and eye beautification may include any one or more or all of thefollowing steps, not necessarily in the order described:

The mouth redness may be increased. In YUV color space this may be doneby multiplying the V value inside the mouth area by a predefined factor(e.g., 1.2).

The teeth may be whitened by slightly increasing the Y component whilereducing the absolute value of U and V components.

The eye white may be brightened and whitened, by slightly increasing theY component while reducing the absolute value of U and V componentsinside the eye white area.

The iris may be improved by stretching the intensity contrast inside theiris area. Also, if the red eye phenomenon is present (which results inan increased V value of the pupil area located inside the iris), a redeye correction algorithm may be applied, as may a golden eye algorithm(see, e.g., U.S. Pat. Nos. 6,407,777, 7,042,505, 7,474,341, 7,436,998,7,352,394, 7,336,821 and 7,536,036, which are incorporated byreference).

In accordance with several embodiments, the quality of portrait imagesmay be improved by doing face, skin and/or face feature enhancement.

Alternative Embodiments

Certain embodiments benefit very advantageously when provided on digitalcamera and especially on a handheld camera-equipped device. Usingspecific data from a face detector, or even a face tracker (with datafrom multiple image frames) can permit the method to performadvantageously. In one embodiment, an enhanced face image may beacquired dynamically from a face tracker module. The use of a PCA todetermine main skin color can be advantageous, as well as using the twoother color space dimensions to determine variation from that color. Themethod may include decorrelating the color space into “primary skin” and“secondary skin”. The use of the “secondary skin” dimensions todetermine “good skin” can be advantageous for skin detection as well. Asmaller image may be used for the detection, while the localizedsmoothing kernel(s) may be applied to the full image, thereby savingvaluable processing resources to great advantage on a handheld device.Two skin maps may be used, including an “exclusive” one combined with an“inclusive” one, and face detection data may also be utilized. Many“skin analysis” and tone/color/contrast and other image adjustmenttechniques may be combined with embodiments described herein, e.g. asdescribed at US published application no. 2006/0204110, which isincorporated by reference. Skin and facial feature detection (eyes,mouth) is advantageously used in facial image enhancement, which mayinclude smoothing, blur, texture modification, noisereduction/enhancement, or other technique for reducing a visual effectof a blemish or blemished region of a face. Wrinkle correction may beeffected within certain embodiments.

In addition, PCA-based “strong” skin detection may be advantageouslyutilized, which enables detection of only those skin tones which aresimilar to those of the face, and may be used to discard other skin-likepatches whose color is yet different from that of the skin (e.g., a wallbehind, light hair, etc.).

The embodiments described herein utilize application of selectivesmoothing which is not to all skin pixels of the face, but only to thosewhich are likely to be or include artifacts (e.g., wrinkles, pimples,freckles etc.). This is very different from global solutions where allfacial skin pixels or the entire face are smoothed and facial non-skinpixels (e.g. mouth, eyes, eyebrows) are sharpened. These embodimentsserve to preserve intrinsic skin textures, while removing unwantedartifacts. For instance, a person's will look their age, thus remainingnatural, while still improving the appearance of the face.

Age Classification

In another embodiment, a processor-based digital image acquisitiondevice is provided, e.g., with a lens and image sensor, a processor andcode for programming the processor to perform a method of enhancingacquisition parameters of a digital image as part of an image captureprocess using face detection within said captured image to achieve oneor more desired image acquisition parameters. Multiple groups of pixelsthat correspond to a face within a digitally-acquired reference imageare identified. Values are determined of one or more attributes of theface. One or more default image attribute values are compared with oneor more of the determined values. The face is classified according toits age based on the comparing of the image attribute values. A cameraacquisition parameter is adjusted based on the classifying of the faceaccording to its age. A main image is captured in accordance with theadjusting of the camera acquisition parameter.

The process may also include generating in-camera, capturing orotherwise obtaining in-camera a collection of low resolution imagesincluding the face, and tracking said face within said collection of lowresolution images. The identifying of face pixels may be automaticallyperformed by an image processing apparatus. Automated processing of theface pixels may be performed based on the classifying.

The camera acquisition parameter may include exposure. The age of theface may be classified as that of a child, baby, youth, adult, elderlyperson, and/or may be determined based on recognition of a particularface. The adjusting of the camera acquisition parameter may includereducing exposure. Fill-flash may be applied to the face inpost-processing. The adjusting of camera acquisition parameter mayinclude optimizing focus on a baby's or child's or youth's face,centering the face, increasing the size of the face, cropping around theface, adjusting the orientation or color of the face, or combinationsthereof, and/or may involve increasing the resolution and/or reducingthe compression of pixels of the face of the baby or child or otherclassification of face.

The face may be tracked over a sequence of images.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention.

In addition, in methods that may be performed according to preferredembodiments herein and that may have been described above, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations, except for those where a particularorder may be expressly set forth or where those of ordinary skill in theart may deem a particular order to be necessary.

In addition, all references cited above and below herein, as well as thebackground, invention summary, abstract and brief description of thedrawings, are all incorporated by reference into the detaileddescription of the preferred embodiments as disclosing alternativeembodiments.

The following are incorporated by reference: U.S. Pat. Nos. 7,403,643,7,352,394, 6,407,777, 7,269,292, 7,308,156, 7,315,631, 7,336,821,7,295,233, 6,571,003, 7,212,657, 7,039,222, 7,082,211, 7,184,578,7,187,788, 6,639,685, 6,628,842, 6,256,058, 5,579,063, 6,480,300,5,781,650, 7,362,368 and 5,978,519; and

U.S. published application nos. 2005/0041121, 2007/0110305,2006/0204110, PCT/US2006/021393, 2005/0068452, 2006/0120599,2006/0098890, 2006/0140455, 2006/0285754, 2008/0031498, 2007/0147820,2007/0189748, 2008/0037840, 2007/0269108, 2007/0201724, 2002/0081003,2003/0198384, 2006/0276698, 2004/0080631, 2008/0106615, 2006/0077261 and2007/0071347; and

U.S. patent applications Ser. Nos. 10/764,339, 11/573,713, 11/462,035,12/042,335, 12/063,089, 11/761,647, 11/753,098, 12/038,777, 12/043,025,11/752,925, 11/767,412, 11/624,683, 60/829,127, 12/042,104, 11/856,721,11/936,085, 12/142,773, 60/914,962, 12/038,147, 11/861,257, 12/026,484,11/861,854, 61/024,551, 61/019,370, 61/023,946, 61/024,508, 61/023,774,61/023,855, 61/221,467, 61/221,425, 61/221,417, 61/182,625, 61/221,455,11/319,766, 11/673,560, 12/485,316, 12/479,658, 12/479,593, 12/362,399,and 12/302,493.

What is claimed is:
 1. A method of enhancing an appearance of a facewithin a digital image, comprising using a processor in: acquiring adigital image of a scene including a depiction of a face, includingcapturing the image using a lens and an image sensor, or receiving saidimage following capture by a device that includes a lens and an imagesensor, or a combination thereof; identifying a face region, wherein theface region is a region of the digital image that includes the depictionof the face; wherein the face region includes less than all of saiddigital image; identifying within the face region one or moresub-regions to be enhanced with localized luminance smoothing; whereineach of the one or more sub-regions includes less than all of said faceregion; applying one or more localized luminance smoothing kernels eachto one of the one or more sub-regions identified within the face regionto produce one or more enhanced sub-regions within the face region,wherein the applying comprises applying the one or more localizedluminance smoothing kernels to luminance data of the one or moresub-regions identified within said face region; wherein the step ofapplying one or more localized smoothing kernels further comprisesapplying, to each particular sub-region of the one or more sub-regions,a localized smoothing kernel configured based, at least in part, on anaverage luminance around a point of interest for that particularsub-region; generating an enhanced image including an enhanced versionof the face region comprising certain original or otherwise processedpixels in combination with pixels corresponding to the one or moreenhanced sub-regions within the face region; and displaying,transmitting, communicating or digitally storing or otherwise outputtingthe enhanced image or a further processed version, or combinationsthereof.
 2. The method of claim 1, wherein the localized luminancesmoothing comprises blurring or averaging luminance data, or acombination thereof.
 3. The method of claim 1, further comprisingapplying one or more localized color smoothing kernels to the one ormore sub-regions, and wherein the one or more enhanced sub-regions ofthe enhanced image further comprise pixels modified from original pixelsof the face at least by localized color smoothing.
 4. The method ofclaim 1, further comprising applying noise reduction or enhancement, orboth, to the one or more sub-regions, and wherein the one or moreenhanced sub-regions of the enhanced image further comprise pixelsmodified from original pixels of the face at least by localized noisereduction or enhancement, or both.
 5. The method of claim 1, whereinenhanced pixels of the one or more enhanced sub-regions compriseenhanced intensities which comprise one or more functions of arelationship between original pixel intensities and local averageintensities within the one or more original or enhanced sub-regions, orcombinations thereof.
 6. The method of claim 1, further comprisingdetecting one or more mouth or eye regions, or combinations thereof,within the face, and identifying and enhancing a natural color of one ormore sub-regions within the one or more mouth or eye regions, includingone or more teeth, lips, tongues, eye whites, eye brows, iris's, eyelashes, or pupils, or combinations thereof.
 7. A device including alens, an image sensor, a processor, and a processor-readable mediumhaving code embedded therein for programming the processor to perform amethod of enhancing an appearance of a face within a digital image,wherein the method comprises: acquiring a digital image of a sceneincluding a depiction of a face, including capturing the image using alens and an image sensor, or receiving said image following capture by adevice that includes a lens and an image sensor, or a combinationthereof; identifying a face region, wherein the face region is a regionof the digital image that includes the depiction of the face; whereinthe face region includes less than all of said digital image;identifying within the face region one or more sub-regions to beenhanced with localized luminance smoothing; wherein each of the one ormore sub-regions includes less than all of said face region; applyingone or more localized luminance smoothing kernels each to one of the oneor more sub-regions identified within the face region to produce one ormore enhanced sub-regions within the face region, wherein the applyingcomprises applying the one or more localized luminance smoothing kernelsto luminance data of the one or more sub-regions identified within saidface region; wherein the step of applying one or more localizedsmoothing kernels further comprises applying, to each particularsub-region of the one or more sub-regions, a localized smoothing kernelconfigured based, at least in part, on an average luminance around apoint of interest for that particular sub-region; generating an enhancedimage including an enhanced version of the face region comprisingcertain original or otherwise processed pixels in combination withpixels corresponding to the one or more enhanced sub-regions within theface region; and displaying, transmitting, communicating or digitallystoring or otherwise outputting the enhanced image or a furtherprocessed version, or combinations thereof.
 8. The device of claim 7,wherein the localized luminance smoothing comprises blurring oraveraging luminance data, or a combination thereof.
 9. The device ofclaim 7, further comprising applying one or more localized colorsmoothing kernels to the one or more sub-regions, and wherein the one ormore enhanced sub-regions of the enhanced image further comprise pixelsmodified from original pixels of the face at least by localized colorsmoothing.
 10. The device of claim 7, further comprising applying noisereduction or enhancement, or both, to the one or more sub-regions, andwherein the one or more enhanced sub-regions of the enhanced imagefurther comprise pixels modified from original pixels of the face atleast by localized noise reduction or enhancement, or both.
 11. Thedevice of claim 7, wherein enhanced pixels of the one or more enhancedsub-regions comprise enhanced intensities which comprise one or morefunctions of a relationship between original pixel intensities and localaverage intensities within the one or more original or enhancedsub-regions, or combinations thereof.
 12. The device of claim 7, whereinthe method further comprises detecting one or more mouth or eye regions,or combinations thereof, within the face, and identifying and enhancinga natural color of one or more sub-regions within the one or more mouthor eye regions, including one or more teeth, lips, tongues, eye whites,eye brows, iris's, eye lashes, or pupils, or combinations thereof. 13.One or more non-transitory computer readable media having code embeddedtherein for programming a processor to perform a method of enhancing anappearance of a face within a digital image of a scene including adepiction of a face captured with a lens and an image sensor, orreceived following capture by a device that includes a lens and an imagesensor, or a combination thereof; wherein the method comprises:identifying a face region, wherein the face region is a region of thedigital image that includes the depiction of the face; wherein the faceregion includes less than all of said digital image; identifying withinthe face region one or more sub-regions to be enhanced with localizedluminance smoothing; wherein each of the one or more sub-regionsincludes less than all of said face region; applying one or morelocalized luminance smoothing kernels each to one of the one or moresub-regions identified within the face region to produce one or moreenhanced sub-regions within the face region, wherein the applyingcomprises applying the one or more localized luminance smoothing kernelsto luminance data of the one or more sub-regions identified within saidface region; wherein the step of applying one or more localizedsmoothing kernels further comprises applying, to each particularsub-region of the one or more sub-regions, a localized smoothing kernelconfigured based, at least in part, on an average luminance around apoint of interest for that particular sub-region; generating an enhancedimage including an enhanced version of the face region comprisingcertain original or otherwise processed pixels in combination withpixels corresponding to the one or more enhanced sub-regions within theface region; and displaying, transmitting, communicating or digitallystoring or otherwise outputting the enhanced image or a furtherprocessed version, or combinations thereof.
 14. The one or morenon-transitory computer readable media of claim 13, wherein thelocalized luminance smoothing comprises blurring or averaging luminancedata, or a combination thereof.
 15. The one or more non-transitorycomputer readable media of claim 13, wherein the method furthercomprises applying one or more localized color smoothing kernels to theone or more sub-regions, and wherein the one or more enhancedsub-regions of the enhanced image further comprise pixels modified fromoriginal pixels of the face at least by localized color smoothing. 16.The one or more non-transitory computer readable media of claim 13,wherein the method further comprises applying noise reduction orenhancement, or both, to the one or more sub-regions, and wherein theone or more enhanced sub-regions of the enhanced image further comprisepixels modified from original pixels of the face at least by localizednoise reduction or enhancement, or both.
 17. The one or morenon-transitory computer readable media of claim 13, wherein enhancedpixels of the one or more enhanced sub-regions comprise enhancedintensities which comprise one or more functions of a relationshipbetween original pixel intensities and local average intensities withinthe one or more original or enhanced sub-regions, or combinationsthereof.
 18. The one or more non-transitory computer readable media ofclaim 13, wherein the method further comprises detecting one or moremouth or eye regions, or combinations thereof, within the face, andidentifying and enhancing a natural color of one or more sub-regionswithin the one or more mouth or eye regions, including one or moreteeth, lips, tongues, eye whites, eye brows, iris's, eye lashes, orpupils, or combinations thereof.